feat(algorithms)!: unify PyTorch runtime contracts - #95

Description

This PR replaces Tributo's divergent PyTorch algorithm integration paths with one versioned contract and one Core-owned Ray Train runtime. TorchRecipe covers Core-owned training loops, RayTorchAdapter covers framework-owned worker loops, and both execute through DistributionStrategy.RAY_TRAIN_TORCH and runtime ID tributo.ray_train_torch under torch_runtime_api_version=1.

The previous TorchTrainingRecipe and TrainingRecipeV2 surfaces both reached the same training implementation, while complex PyTorch algorithms bypassed that path by constructing their own TorchTrainer. The new boundary makes TorchPolicy.execution_plan the single source of truth and assigns Trainer construction, role routing, collective alignment, checkpoint transport, execution evidence, and Bundle publication to Core; algorithm Wheels retain model, loss, metric, framework-adaptation, and artifact-signature responsibilities.

Key changes:

• Add versioned TorchPolicy, single-stage and component-stage execution plans, role-routed datasets, typed runtime and stage contexts, and public TorchRecipe and RayTorchAdapter SPIs.
• Add the unified tributo.ray_train_torch runtime with Torch-only preflight, Core-owned TorchTrainer orchestration, exact split and bounded replication routing, and explicit final-stage selection.
• Add normalizer-aware loss accumulation, composite-loss reducer invocation, explicit metric reducers, rank-aligned zero contributions for optional evaluation roles, and typed execution evidence bound to the resolved Policy and plan.
• Add stage-scoped run identities and Core-owned checkpoint descriptors while delegating Worker failure retry and checkpoint storage to Ray; Runtime API v1 rejects cross-Run recovery and transports only invocation-local predecessor Stage checkpoints.
• Replace ray-torch-recipe-v1 with ray-torch-v1, split Recipe reconstruction from Adapter export-source handling, and route both through the existing BundleExportService.
• Add an installed-distribution conformance CLI and fixed identity manifest for source-free Wheel verification.
• Migrate Core fixtures, public API inventory, architecture documentation, official algorithm identities, migration guidance, and regression tests to the new PyTorch contract.

This is intentionally one atomic implementation change because the public Policy, SPI, Runtime, Checkpoint, Evidence, and Source Provider identities must move together; merging an intermediate subset would leave installed PyTorch descriptors without a valid execution or export path.

Related issues

• Follows up on feat(training): add Ray-first PyTorch recipe execution #73, which introduced the previous Ray-first PyTorch Recipe path.
• Builds on the official algorithm Wheel boundary from feat: complete distributed algorithm wheel ecosystem #81 and the 37-entry-point certification baseline from test: certify official distributed algorithms #85.

Additional information

Cross-repository coordination

• Core implementation: cdb247c.
• Companion algorithm migration: 4edcc5f, already present on tributo-algorithms/master.
• Candidate Core Wheel provenance: Pending. No candidate Wheel has been built from cdb247c; existing candidate Wheels predate the final Core commit and are not valid evidence for this PR.
• The source merge order is therefore algorithm migration first and this Core PR second. The Core PR must not merge until joint validation succeeds; package publication order is handled separately from this source merge.

Breaking changes and migration

• Remove TorchTrainingRecipe, TrainingRecipeV2, AlgorithmBuilder.from_torch_recipe(), and AlgorithmBuilder.from_training_recipe_v2().
• Remove the PyTorch-specific legacy Recipe strategy, execution mode, topology, runtime identity, and ray-torch-recipe-v1 source-provider identity.
• Replace those entry points with TorchRecipe, RayTorchAdapter, AlgorithmBuilder.from_torch(), AlgorithmBuilder.from_torch_adapter(), RAY_TRAIN_TORCH, tributo.ray_train_torch, and ray-torch-v1.
• Require PyTorch algorithm Wheels to declare TorchPolicy, the execution plan, role routing, metric reducers, state layout, and the independent Torch Runtime API version.
• Reject legacy PyTorch recovery options and checkpoints; Torch Runtime API v1 does not support cross-Run recovery, and no compatibility reader or automatic upgrade is provided.
• Rename the official PyTorch entry points dnn.recipe_v2, pu.recipe_v2, lstm_classifier.recipe_v2, and gru_classifier.recipe_v2 to their algorithm identities and remove the .recipe_v2 suffix from the LSTM and GRU implementation IDs.

The companion tributo-algorithms migration must be released in coordination with this Core change. Existing PyTorch Wheels implementing the removed interfaces are not compatible with the new runtime.

Compatibility boundary

• The breaking boundary is limited to PyTorch algorithm contracts, identities, checkpoints, and the Torch Source Provider.
• MapReduce, Joblib estimator, parallel ensemble, iterative optimization, XGBoost collective, and non-PyTorch framework-native strategies retain their existing contracts and runtime IDs.
• Existing Bundle Manifest, ONNX Runtime, Ray Data batch inference, Ray Serve, Data Provider/Binding, WriteGateway, ResultSink, MLflow, OperationStore, Outbox, Hook, Broker, and Streaming contracts are not migrated.
• Previously published ONNX Bundles remain readable; recovery checkpoints are not Bundles and follow the explicit incompatibility described above.
• Core imports and descriptor-only discovery remain free of eager Torch, PyG, Transformers, and algorithm implementation imports.
• No third-party dependency is added or changed.

Local static and changed-test validation is complete; Docker Ray IT, companion Wheel conformance, and external cluster gates remain pending before merge.

Validation completed for commit cdb247c

• uv run --locked --no-sync python scripts/pr-precheck.py — format, dependency, CI-policy, API-stability, suppression, commit, hygiene, changed-test, and CI-parity layers passed; the run identified one documentation spelling issue, which was fixed without changing runtime or test code.
• Documentation spelling passed after that one-word fix, and uv run --locked --no-sync python scripts/pr-precheck.py --skip-tests then passed every non-test layer without repeating the completed changed-test run.
• Core affected regression suite — 1016 passed, 9 skipped, 113 deselected.
• Torch Runtime public-contract suite — 34 passed, 2 skipped.
• Companion algorithm repository full unit suite on revision 4edcc5f — 220 passed.
• Core source mypy and companion changed-source mypy checks — passed.
• Pre-commit Ruff format and lint hooks — passed.
• git diff --check — passed.
• The pre-check reported 116 non-blocking Python Safety warnings; these are primarily function-local Torch or Ray imports that preserve lightweight module loading. Its four legacy projection warnings are pre-existing test callers outside this PyTorch migration.

Pending validation before merge

• Pending: Run the repository's full non-external unit suite for Python 3.12 and 3.13 through the PR CI matrix.
• Pending: Run source-free 37-entry-point conformance in an external clean environment using the recorded Core and 15 algorithm Wheel SHA-256 values and verified non-editable installation paths.
• Pending: Run the final distributed-algorithm-cluster manual gate on exact Core revision cdb247c and algorithm revision 4edcc5f to cover multi-worker Ray Jobs training, exact role routing, component Stage dependencies, Tune fit-only checkpoint behavior, Bundle publication, and Ray Data inference without skipped tests.
• Pending: Review the additional manual-external suites selected by scripts/ci_test_plan.py for the pyproject.toml, runtime, export, inference, and documentation changes, and record each suite as passed or explicitly not required with a concrete scope reason.
• Pending: Do not issue GPU, NCCL, PyG, Transformers, FSDP, DeepSpeed, or TorchRec support claims without separately approved environment-specific gates.

Fixed-Wheel and cluster results remain pending and will not be inferred from local unit evidence.

Reviewer focus

• Verify that every shared Builder, Planner, Dispatcher, Receipt, Support, and Bundle change is selected only by a Torch discriminant and does not alter the 24 non-PyTorch algorithm identities, contracts, routing, errors, or lifecycle.
• Verify that TorchPolicy.execution_plan is the only stage and routing declaration and that neither Recipe nor Adapter can create a nested Trainer or provide a competing execution plan.
• Verify the normalizer-weighted accumulation formula, composite reducer trust boundary, collective ordering, AMP unscale and gradient scaling order, zero-normalizer rejection, and metric reducer mapping.
• Verify Torch-only preflight, Stage run identity, Ray-owned Worker failure retry, predecessor Stage checkpoint transport, and explicit cross-Run recovery rejection.
• Verify component Stage evidence completeness and dependency Checkpoint sources, and confirm that only final_stage_id can supply the exported Bundle.
• Verify the ray-torch-v1 Recipe and Adapter export branches, Artifact Plan binding, required-artifact failure behavior, and composition_digest propagation.

Non-goals

• This PR does not implement or claim FSDP, DeepSpeed, TorchRec, Lightning, Hugging Face Trainer, or native PyTorch serving support.
• This PR does not add a second DataLoader, distributed checkpoint store, cluster manager, deployment controller, or Bundle format.
• This PR does not remove global inference or serving compatibility paths or refactor unrelated Data, MLflow, ResultSink, OperationStore, Broker, or Streaming modules.